Determining the Degree of Generalization Using an Incremental Learning

Any Learning Machine (LM) trained with examples poses the same problem: how to determine whether the LM has achieved an acceptable level of generalization or not. This work presents a training method that uses the data set in an incremental manner such that it is possible to determine when the behavior displayed by the LM during the learning stage truthfully represents its future behavior when confronted by unseen data samples. The method uses the set of samples in an efficient way, which allows discarding all those samples not really needed for the training process. The new training procedure, which will be called “Incremental Training Algorithm”, is based on a theoretical result that is proven using recent developments in statistical learning theory. A key aspect of this analysis involves identification of three distinct stages through which the learning process normally proceeds, which in turn can be translated into a systematic procedure for determining the generalization level achieved during training. It must be emphasized that the presented algorithm is general and independent of the architecture of the LM and the specific training algorithm used. Hence it is applicable to a broad class of supervised learning problems and not restricted to the example presented in this work.